LJ These slides/notes represent only part of the course, and were 
accompanied by face-to-face explanations on white-board and 
additional topics / learning materials. 


LI In preparation of these slides | have also benefited from various 
books and online material. 


LI Some of the slides contain animations which may not be visible 
in pdf version. 


LJ Corrections, comments, feedback may be sent to 
httos://www.linkedin.com/in/naveedrazzaqbutt 
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«Me 


° You 
¢ The Course 
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Important Business!! 


¢ 75% attendance is mandatory! 


¢ Textbooks 
¢ Peyton Z. Peebles, JR, Probability, Random Variables and Random Signal Principles, 4th 
Edition, McGraw-Hill, 2002 
¢ D.C. Montgomery & G. C. Runge, Applied Statistics and Probability for Engineers, 6th Edition, 
Wiley, 2013 


¢ Contact 


¢ nbutt@ju.edu.sa 
¢ office: 1140 
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Weightage 


Learning Plan 


@ Final @Mid1 MMid2 MQuizzez Project 


¢ Lectures 
¢ Help discover and grasp new concepts 

¢ Quizzes (six) 
¢ Help prepare/revise each week’s concepts 
¢ Keep you from lagging behind in course 


¢ Course Project 
¢ Helps learn independent work & presentation 
¢ Prepares for final year project 

¢ Exams (Mid-1, Mid-2, Final) 


¢ Help prepare entire course material 
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Course project 


Assessment Plan 





Mid-Term 





Final 
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Course Ends 


In this course we will see ... 


¢ What is “probability” and what are its fundamental principles 
¢ How can probabilities be assigned to different kinds of events 
¢ What are the different types of random variables 


¢ What is meant by a probability distribution and what are the 
different common probability distributions 


¢ How can we see the link between various random events 
¢ What are random processes and some of their characteristics 


Course Learning Objectives (CLOs) 
a 


CLO 1 Cognitive Skills Calculate probabilities of events, joint probabilities, HW, Quiz, Mid, 
conditional probabilities using set operations and definition of Final 
probability. Justify valid and invalid probability assignments, 
and independence of events. 

CLO 2 Cognitive Skills _ Calculate probability mass function parameters, moments and HW, Quiz, Mid, 
functions of discrete single and multiple random variables. Final 


CLO 3 Cognitive Skills _ Calculate probability density function parameters, moments HW, Quiz, Mid, 
and functions of continuous single and multiple random Final 
variables. 

CLO 4 Cognitive Skills . Analyze random processes and effects of linear systems on HW, Final 
random processes 

CLO 5 Communication Demonstrate the ability to research a topic related to Project 
probability and formally present the results Presentation 
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What is “probability” ? 
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What is “probability” ? 


¢ Probability is a “lack of knowledge”! 
¢ We know you are here today. We are sure. 


¢ But will you be here in the next lecture? We are not sure anymore! There is 
now a “lack of knowledge” 


yy Mt yy Mt 


* “perhaps”, “maybe”, “probably” 


Why do we sometimes lack knowledge ? 


What An Electron Isn't 
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Why do we sometimes lack knowledge ? 


¢ Future 
¢ A dice you haven't rolled yet 
¢ How can we know which number it will show! 
¢ Too hard to collect all the information 
¢ Which places did you visit today? 
¢ It may be possible to have a drone camera follow you all the time. Then we will not have 
“lack of knowledge” about places you go to. But this is too hard a thing to do. 
¢ Quantum randomness 


¢ Where's the electron? 


¢ According to current consensus, processes and properties at quantum level are 
probabilistic by their very nature. 


Why study probability ? 
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Why study probability ? 


¢ Height of the next student who enters the room. 


Why study probability? 


¢ Height of the next student who enters the room. 





Number of People (1000s) 


0 . 
140 145 150 155 160 165 170 175 180 185 190 195 200 


Height (cm) 
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Why study probability ? 


¢ Height of the next student who enters the room. 
¢ There is lack of knowledge about it! 


¢ But that lack of knowledge is not “absolute” 


¢ We do know something about the heights of humans and can make some “guesses” 
based on whatever information we have (based on observation, experience, statistics) 


¢ Such guesses can help us design the height of the doorway (for instance). 


Probability theory helos us make sense of an 
uncertain world! 


¢ It helps us make smart guesses about uncertain events 


¢ Based on the smart guesses we can plan, design, or take steps to 
better control the situation 


Questions??? Thoughts? ? 


° 


e 
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A set of possibilities ... 
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A set of possibilities ... 


¢ Suppose we roll a dice 


¢ Before we can say something about the possibility of seeing a 
particular number, we have to determine what is possible and what is 
not. 


¢ We need to define a set of possibilities for the dice rolling! 


“Set” of possible 
outcomes 


“Set” of impossible 
outcomes 
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A set of possibilities ... 


“Set” of possible 
outcomes 


a 


Universal Set or Sample Space 
(S) for the die rolling experiment 
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“Probability” and “Sets” go hand-in-hand! 


¢ Understanding sets is an important part of understanding probability. 


Definition: Set 


¢ Aset is a collection of objects 
¢ The objects are called “elements” of the set 


A = {a,b,c,d} 
aeA 


e€gA 


My students ... 


Dr. Naveed’s 
students 


A 





Students in 


EE302 Students in 


EE328 
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Basic set operations... 


Dr. Naveed’s 


students . 
(assuming | am teaching only two courses this term) 


B = { my students in EE302 } 
C= { my students in EE328 } 
B U C= my students who are in EE302 or in EE328 = 

all of my students this term 

BOC = my students who are in both EE328 and 

EE302 (multicolor) 
B— C= my students who are in EE302 but not in 
EE328 (yellow) 
C—B = my students who are in EE328 but not in 
EE302 (red) 





Students in 
EE302 Students in 
EE328 
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Dr. Naveed’s 
students 


We also see that: 


DCC (all elements of D are also in C) 





BUC=C UB 
BNC=CNB 
B—-C +C-—B 


A —(B U C)=@ (empty set) 


Students in 


EE302 Students in 


EE328 
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Set Operation Venn Diagram Interpretation 


A U B, is the set of all 
Union values that are a member 
of A, or B, or both. 


AB, is the set of all 
intersection values that are members 
of both A and B. 
A\ 8B, is the set of all 
: values of A that are not 
Difference 
members of B 
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Some formal definitions ... 


¢ An “empty” set is a set with no elements 
°O=}33 
¢ Two sets A and B are “mutually exclusive” or “disjoint” if they have 
no common elements 
“-ANB= @ 
¢ A “universal set” is the largest or all-encompassing set of objects 
under discussion (usually denoted S or U) 


¢ The complement of a set A is a set of all elements not in A 
°A=S-A 


Some formal definitions ... 


|’” 


¢ Two sets are “equal” if they are both subsets of each other (i.e., they 
have exactly the same elements) 
-ifAcCBandB CAthenA=B8B 


Set Operations and Venn Diagrams 





SetA A’ the complement of A 
A B 
A and Bare disjoint sets Bisproperr BcA 
subset of A 
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Three Ways of Defining a Set 


¢e Aset may be defined in three ways 
¢ Tabular method: we list (write) all the elements of the set 
¢ Rule method: we describe a rule for elements of the set 
¢ Operation method: we define the set as a result of some operation 


Tabular f= {Abdallah, Mishari, Yazeed, Bandar, Basil, Badr, Abdullateef, Samir} 
Rule A = {students in EE302 second semester 2019} 


Operation C=B-A 
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Notation 


¢ Notation for Union and Intersection when more than two sets are 
involved 
¢ Recall that that when we add several variables we can denote the sum by 
using the symbol “sigma” 
Fl 
SX) =H Xp HX HGH... X). 
=| 


¢ Similarly, union and intersection of several sets can be denoted as 


N 
C=A4,UA,U---UAy=| JA, 


n= | 

VV 
D= AMAL Aye () A, 

i1- | 
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Some properties of sets... 


¢ Commutative, distributive and associative properties of union and 
intersection. 


ANB=BNA AN(BUC)=(ANB)U(ANC) 
AUB=BUA AU(BNC)=(AUB)N(AUC) 


(AUB)UC=AU(BUC)=AUBUC 
(ANB)NC=AN(BNC)=ANBNC 
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Some properties of sets... 


¢ De Morgan’s Laws 


(AUB) =ANB 


‘ 
(AN B)=AUB 
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Some properties of sets... 


¢ Duality Principle 
e “if in an identity we replace unions with intersections, and intersections with 
unions, and also replace universal set (S) with empty set (@), and empty set 
with universal set then identity remains valid”. 


AN(BUC)=(AN B)U(ANC) 
If one is valid, then so is the 
tl other! 


AU(BNC)=(AUB)N(AUC) 
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Example Problems 
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Questions??? Thoughts? ? 


° 


e 
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In the last lecture, we saw that ... 
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Quick Revision (1) 


; : A = {a,b,c, da} 
¢ A set is a collection of objects ned 
¢ Objects in a set are called elements of the set i 
¢ We often need sets to define probabilities 


¢ The set of all possible outcomes in a situation is called the universal 
set or sample space (denoted S) 


¢ A set with no elements is called an empty set (denoted @ or {}) 


e We saw how some sets are related to each other 
¢ Subset: when all the elements of set A can be found insetB (A & B) 
¢ Equality: when two sets have exactly the same elements (A = B) 


Set = collection of objects 





Students in 
EE302 
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Quick Revision (1) 


: ; A = {a,b,c, da} 
¢ A set is a collection of objects ned 
¢ Objects in a set are called elements of the set i 
¢ We often need sets to define probabilities 


¢ The set of all possible outcomes in a situation is called the universal 
set or sample space (denoted S) 


¢ A set with no elements is called an empty set (denoted @ or {}) 


e We saw how some sets are related to each other 
¢ Subset: when all the elements of set A can be found insetB (A & B) 
¢ Equality: when two sets have exactly the same elements (A = B) 


Quick Revision (1) 


: _ A = {a,b,c, da} 
¢ A set is a collection of objects ned 
¢ Objects in a set are called elements of the set i 
¢ We often need sets to define probabilities 


¢ The set of all possible outcomes in a situation is called the universal 
set or sample space (denoted S) 


¢ A set with no elements is called an empty set (denoted @ or {}) 


e We saw how some sets are related to each other 
¢ Subset: when all the elements of set A can be found insetB (A & B) 
¢ Equality: when two sets have exactly the same elements (A = B) 


A set of possibilities ... 


“Set” of possible 
outcomes 


a 


Universal Set or Sample Space (S) 
for the die rolling experiment 
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Quick Revision (1) 


: _ A = {a,b,c, da} 
¢ A set is a collection of objects ned 
¢ Objects in a set are called elements of the set i 
¢ We often need sets to define probabilities 


¢ The set of all possible outcomes in a situation is called the universal 
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Quick Revision (1) 


: _ A = {a,b,c, da} 
¢ A set is a collection of objects ned 
¢ Objects in a set are called elements of the set i 
¢ We often need sets to define probabilities 


¢ The set of all possible outcomes in a situation is called the universal 
set or sample space (denoted S) 
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Quick Revision (1) 


; A = {a,b,c, da} 
¢ A set is a collection of objects wen 


¢ Objects in a set are called elements of the set eon 


¢ We often need sets to define probabilities 


¢ The set of all possible outcomes in a situation is called the universal 
set or sample space (denoted S) 


¢ A set with no elements is called an empty set (denoted @ or {}) 


e We saw how some sets are related to each other 
° Subset: when all the elements of set A can be foundin set B (A S B) 
¢ Equality: when two sets have exactly the same elements (A = B) 
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My students... 


Dr. Naveed’s 
students 


A 


BCA 





Students in 


EE302 Students in 


EE328 
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(all elements of B are also in A) 


11 


Quick Revision (2) 


We saw how we can check equality of two sets 
° if AC BandB CAthenA=B 


* We learned that two sets are completely different (mutually exclusive or disjoint) 
if they have no elements in common 
¢ Check for mutual exclusiveness: A NB= @ 


We also saw some operations we can perform on sets 
¢ Union: A U B take all the elements of the sets A and B 
¢ Intersection: AN B = take only those elements which are common between A and B 
¢ Difference: B — A = remove from B all those elements which are also present in A 


¢ Complement: A = S — A (remove all elements of set A from universal set S) 


¢ Finally, we discussed some useful properties of set operations 
¢ Commutative, Distributive, Associative 
¢ De Morgan’s Laws 
¢ Duality Principle 
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Quick Revision (2) 


We saw how we can check equality of two sets 
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Quick Revision (2) 


We saw how we can check equality of two sets 
° if AC BandB CAthenA=B 
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¢ Check for mutual exclusiveness: A NB= @ 


We also saw some operations we can perform on sets 
¢ Union: A U B take all the elements of the sets A and B 
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¢ Difference: B — A = remove from B all those elements which are also present in A 


¢ Complement: A = S — A (remove all elements of set A from universal set S) 


Finally, we discussed some useful properties of set operations 
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¢ De Morgan’s Laws 
¢ Duality Principle 
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My students... 


Dr. Naveed’s 
students 


A 


B = { my students in EE302 } 
C= { my students in EE328 } 
B U C= my students who are in EE302 or in EE328 = 

all of my students this term 

BOC = my students who are in both EE328 and 

EE302 (multicolor) 
B— C= my students who are in EE302 but not in 
EE328 (yellow) 
C—B = my students who are in EE328 but not in 
EE302 (red) 





Students in 


EE302 Students in 


EE328 
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Quick Revision (2) 
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| ANB=BNA 
Commutative 
Property AUB=BUA 


Distributive AN (BU C) = (A MO B)U (A () é,) 
Property AU(BNC)=(AUB)N(AUC) 


Associative (AUB)UC=AU(BUC)=AUBUC 
i (ANB)NC=AN(BNC)=ANBNC 
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\| 
pa | 


| 
cs. oe 
Sor Gy 


De Morgan’s (AUB 
Laws 


a 


) 
(ANB) =. 


AN (B UC ) — (A () B) U (A M C) “if in an identity we replace unions with 


Duali intersections, and intersections with 
; un 2 tl unions, and also replace universal set (S) 
rinciple with empty set (Q), and empty set with 


AU ( BN C) 1 ( 4u B) A ( ria C) universal set then identity remains valid”. 
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Today ... 


¢ We will go through some examples 


¢ And, see how we can assign probabilities to random events 
¢ What is the process? 
¢ What are the conditions? 


Definition: Random Event 


¢ An outcome (or result) we are not sure about. 


¢ Example: when we toss a coin the occurrence of a head or tail is a random 
event. 


Defining “Probability” of a Random Event 


P(A) = # of ways A can occur 
: total # of outcomes 
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A set of possibilities ... 


“Set” of possible 
outcomes 


a 


Universal Set or Sample Space 
(S) for the die rolling experiment 
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How do we assign probabilities to random 
events? 


¢ What is the process? 
e What are the conditions? 


Let’s play ... 


¢ Assume there is a bag with only one ball in it which is yellow 


¢ You put your hand in the bag, and without looking, take a ball out and note its 
color 


¢ What are the possible outcomes? 


Set of all the we >= {Yellow} 


possible outcomes 
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P(A) = 


# of ways A can occur 


Let’s play ... 


¢ What is the probability that the ball you draw is yellow? 


Outcome of our 


| A = {Ball drawn is Yellow} = “rt 
Interest 


Set of all the S = {Yellow} Universal Set 
possible outcomes 


total # of outcomes P(A) = P(yellow ball) = 1 100% (sure) 
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Let’s play ... 


¢ Now assume there are two balls in the bag, yellow and red 


¢ You put your hand in the bag, and without looking, take a ball out and note its 
color 


¢ What are the possible outcomes? 


Set of all the we >= tYellow, Red} 


possible outcomes 
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Let’s play ... 


¢ What is the probability that the ball you draw is yellow? 


Outcome of our 


| A = {Ball drawn is Yellow} = “rt 
Interest 


Set of all the 


S = {Yellow Red} Universal Set 
possible outcomes t 


# of ways A can occur 
P(A) = ——— 


total # of outcomes P(A) = P(yellow ball) = 1/2 50% chance 
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Let’s play ... 


¢ What is the probability that the ball you draw is yellow? 


Outcome ofour A = {Ball drawn is white} =" 


interest 


Set of all the S= {Yellow, Red} Universal Set 


possible outcomes 


# of ways A can occur 5 
jo P(A) = P(white ball) = 0 impossible 


total # of outcomes 
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Let’s play ... 


¢ What is the probability that the ball you draw is yellow or red? 


Outcome ofour A = {Ball drawn is Yellow or Red} <r" 


interest 


Set of all the S= {Yellow, Red} Universal Set 


possible outcomes 


p(4) = 2 Ot Ways A can occur P(A) = P(red or yellow ball) = 1 


total # of outcomes 





EE 302 - Probabilistic Methods in Electrical Engineering / Dr. Naveed R. Butt @ Jouf University 


Let’s play ... 


e Assume we throw a die, and note the number it shows 
¢ What are the possible outcomes now? 


Set of all the a ={12345 6} ‘ 


possible outcomes 


EE 302 - Probabilistic Methods in Electrical Engineering / Dr. Naveed R. Butt @ Jouf University 


Let’s play ... 

We use this “size” or element 

counting approach only if all 
events are equally likely 


¢ What is the probability that it shows 4? (equally likely = have equal 


chance of happening) 


Outcome of our A= {shows 4} Event 


interest 

Set of all the S= {1 2345 6} Universal Set 4 e @ 

possible outcomes ‘ eo & 
a @ 


Fe ac ae P(A) = 1/6 = size(A)/size(S) 


total # of outcomes 
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Let’s play ... 


¢ What is the probability that it shows an even number? 


Outcome of our A= {shows 2, 4 or 6} Event 


interest 


possible outcomes 


Set of all the S= {1 2345 6} Universal Set 4 


p( a) = Ot ways A can occur P(A) = 3/6 = size(A)/size(S) 


total # of outcomes 
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Let’s play ... 


¢ What is the probability that it shows a number from 1 to 6? 


Outcome of our A= {shows 1,2,3,4,5,or 6} Event 


interest 


possible outcomes 


Set of all the S= {1 2345 6} Universal Set 4 


pca) == was Remo’ P(A) = 6/6 = size(A)/size(S) 


total # of outcomes 
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Assigning probabilities is a five-step process 


¢ Step 1: Define the experiment 
e “let’s roll.a die” 


¢ Step 2: Define all the possible outcomes of the experiment, i.e., define the 
universal set that applies to your experiment 
© S={123456} 


¢ Step 3: Define what outcome you would like to see (we call it “event”) 
¢ A= {die show even number} = {2 4 6} 


: Step 4: Assign probability to the event P(A) = # of ways A can occur 
e P( A) =3 /6 =¥, | total # of outcomes 


¢ Step 5: Make sure that the assigned probabilities “make sense” 
¢ The Three Conditions (discussed later today)! 
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Example Problems 
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Event Types: Equally Likely 


¢ Equally Likely 
¢ Events that have equal chances of occurring are called “equally likely” events 
e E.g., when tossing a coin, there are equal chances of H and T 
¢ Sometimes we use common sense to claim that some events are equally likely 
e At other times we may be given equal probabilities assigned to events, and consequently 
may claim them to be equally likely 


Event Types: Mutually Exclusive 


¢ Mutually Exclusive 
e Events that cannot occur at the same time are called mutually exclusive 
¢ E.g., when tossing a coin a head (H) and tail (T) cannot occur at the same time 


¢ In sets, we check it this way: if the intersection of two events is empty set then they are 
mutually exclusive 


Two events, denoted as E, and E,, such that 
E, fi E, aa O 


are said to be mutually exclusive. 
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Event Operations: Probability of a Union 


P (A or B) 
P(A UB) = P(A) + P(B) — P(ANB) 


If A and B are mutually exclusive events, 


P(A U B) = P(A) + P(B) 
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Event Operations: Probability of a Union 


P(A UBUC) = P(A) + P(B) + P(C) — P(ANB) 
~ PAN C) — (BNC) + PANBNOC) 
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Event Operations: Probability of Intersection 


P(A and B) = P(ANB) 


¢ When we assign probabilities to two or more events happening at the 
same time, we call it joint probability 


¢ More on this later... 


Event Operations: Probability of Complement 


P (not E) 


P(E’) = 1 — P(E) 
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Assigned probabilities must satisfy three conditions! ! 
¢ Condition 1: Assigned probabilities should not be negative or greater than 1. 
0 < P(E) <1 
¢ Condition 2: Universal set should cover all the possible outcomes 
P(S) = 1 


¢ Condition 3: Probabilities assigned to mutually exclusive events should make 
sense 


For two events E, and E, with E; M Ey = © 


P(E, U Ey) = P(E,) + P(E>) 
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Let’s check ... 


S={123456} Universal Set 


A = {1}, B= {2}, C={3}, D= {4}, E= {5}, F= {6}, G={246},... Events 


Assigned Y 
P(A) = 1/6 , P(B) = 1/6, P(C) = 1/6, P(D) = 1/6, P(E) = 1/6, P(F) = 1/6, P(G) = 1/2 probabilities are 
not negative 
P(S) = P (die shows 1, 2, 3, 4,5, or 6) =1 Defined universal set covers VY 
all the possible outcomes 
P(A U B) = P(A) + P(B) 
P(A U BU C) = P(A) + P(B) + P(C) Mutually exclusive events i TZ 
satisfy the third condition 7 ' * @ 
etc... \ * * 
a @ 
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Example Problems 
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Previously, we played dice... 


We were assigning probabilities to 
; : individual events 
¢ We throw a die, and note the number it shows 


¢ What are the possible outcomes? 


¢ What is the probability that it shows 4? 


Outcome of our A= {shows 4} Event 


interest 


ee Shale S — {1 2 3 4 5 6} Universal Set 4 a ®@ 
possible outcomes | * * 
@ @ 


P(A) = 1/6 = size(A)/size(S) 


EE 302 - Probabilistic Methods in Electrical Engineering / Dr. Naveed R. Butt @ Jouf University 


Sometimes we need to assign probabilities to 
multiple events! 


¢ Let’s say its 8 am, and let’s define two events 


A = {Ali is in lecture} Event one 
B = {Ali is sleeping} Event two 


¢ What is the probability that Ali is in lecture and sleeping? 


P(A and B) 


Joint Probability 


¢ When we assign probabilities to two or more events happening at the 
same time, we Call it joint probability 


P(A and B) 


e Using sets, we can define the joint probability of two events as 


P(A and B) = P(AN B) 


) e 
Let’s play again... 
¢ We throw a die, and note the number it shows 
¢ Let us define two events of interest 


A = {shows odd number} Event one 


B = {shows even number}  £vent two 
S= {1 2345 6} Universal Set 
P(A and B) = P(AN B) =0_— How? 


A 1 B = {} (empty) 





P(A N B) = size(A N B)/size(S) = 0/6=0 
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Let’s play again ... 
e We throw a die, and note the number it shows 


C = {shows 1, 2 or 3} Event one 


D = {shows 2, 3, or 4} Event two 


S= {1 2345 6} Universal Set 


P(C and D)=P(CN D)=1/3 ow? - 


—J tk 
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CND={23} 





P(C N D) = size(C N D)/size(S) = 2/6=1/3 





Sometimes knowledge of one random event can help us 
assign probability to another random event 


A = {Ali is in lecture at 8 am} Event one 
B = {Ali is sleeping at 8 am} Event two 


¢ Suppose | tell you that Ali is in the lecture at 8 am. Now what are the 
chances that he is sleeping? 


P(B given A) 


Let’s toss a coin ... 


¢ Suppose | toss a coin but do not show you the result 
¢ What is the probability that it shows Head? 
¢ A= {shows Head} 
¢ S = {H T} (universal set) 
© P(A) =% 
¢ Suppose now that | tell you that it shows Tail 
¢ Now what is the probability that it shows Head? 


Let’s toss a COin... 


Situation before additional information 


A = {shows Head} 


S={HT} Universal Set 


P{A} = 1/2 
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Situation after additional information 


A = {shows Head} 


“H” is no longer in 


S= {h T} a «te set of 


possibilities (it has 
become impossible) 


P{A| new information} = O ————— 


Another example ... 


¢ Suppose | write a number from 1 to 4 on a piece of paper but do not 
show you what | wrote 
¢ What is the probability that | wrote 3? 
¢ A= {wrote 3} 
¢ S={1 2 3 4} (universal set) 
¢ P(A) = size(A)/size(S) = % 
¢ What is the probability that | wrote an odd number? 
¢ B= {wrote 1 or 3} 
¢ S={1 2 3 4} (universal set) 
¢ P(B) = size(B)/size(S) = 2/4 =% 
¢ Suppose now that | tell you that | wrote an odd number. 
¢ Now what is the probability that | wrote 3? 


Another example ... 


Situation before additional information Situation after additional information 


A = {wrote 3} A = {wrote 3} 


. B = {wrote odd number} 
S={1234} Universal Set 


PA} = % Snew = {1X3} vias 


Universal Set 


P{A given B} = P{A|B} = 1/2 
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Conditional Probability 


¢ When we assign probabilities to an event assuming (or knowing) that 
another has happened, we call it conditional probability 


P(A given B) also written as P(A | B) 


¢ Using sets, we define the conditional probability as 


Joint probability 
P(A NB yo of Aand B 


P(B) Important assumption : P(B) # 0 


P(A | B) = 


Let’s play again ... A = {shows 1, 2 or 3} 
B = {shows 2, 3, or 4} 
S={123456} 


e We throw a die, and note the number it shows 


_P(ANB) —_ 1/3 _ 
P(A | B)=—“peBy =a = 2/8 
[on eee}, 


P(A NM B) = size(A N B)/size(S) = 2/6=1/3 ° 8 


P(B) = size(B)/size(S) = 3/6=1/2 
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Example Problems 
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Independence : when knowledge of one event 
does not change the probability of another 


¢ Sometimes knowledge of one event does not affect the probability of 
another event. 
¢ In such a situation, we say that the two events are statistically independent 
e Example: knowing what time | woke up this morning will not affect the result 
of a coin toss 


¢ How can we check independence? 


_ Test 1: If the conditional probability remains 
P(A | B) =r (A) unchanged, then the two events are independent 


P(A 1 B) = P(A) P(B Test 2: If the joint probability is simply the product of 
( ( ( individual probabilities, then the two events are independent 


When we know events are independent... 


¢ An important consequence of knowing that two events are 
independent is that we can use the following substitutions where 
needed 


P(A | B) = P (A) 


P(A M B) = P(A) P(B) 


Can we check independence of more than 
two events? 


e Yes! 


¢ For three events to be statistically independent, all their combinations 
should satisfy test 2, i.e., they must satisfy all of the following 


P(A, M Ax) = P(A,)P(A2) 
P(A, N A3) = P(A) P(A3) 
P(A 1. A3) = P(A2)P(A3) 
P(A, NA, 1 A3) = P(A1)P(A2)P(A3) 


¢ Similarly, for N events to be statistically independent, all their combinations 
should satisfy test 2. 
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Example Problems 
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Two formulas that help us in assigning 
probabilities 


Events B,, must satisfy 
these conditions 


N 
Total Probability P(A) = »F F (A|B,)/ (B,,) N 
= | B,, = § 
n=l 
Bayes’ Theorem P(B,|A) = ae Bin Bele 


i.e., these should be 
mutually exclusive and 
should cover all S. 
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Counting 


¢ In assigning probabilities we often need to “count” how many ways 
something can happen 
¢ Toss a coin twice, and define event 
e A= {we get at least one Tails} 
¢ S={HH HT THTT} 
¢ Count total number of ways two coin tosses can result: four (HH, HT, TH, TT) 
¢ Count number of ways A happens: three (HT , TH, TT) 
¢ P(A) = 3/4 


Counting: Two Questions to Ask 


¢ Question1: Does order matter? 
° Question2: Is repetition (replacement) allowed? 


Question1: Does order matter? 
Question2: Is repetition allowed? 


Game: choose two of these and | will give 10 


Riyals to each. Cannot choose same person twice. va? 

Here: 

(1) ORDER DOESN’T MATTER 

(2) REPETITION NOT ALLOWED 


Possible choices: three 
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Question1: Does order matter? 
Question2: Is repetition allowed? 
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Game: choose two of these. | will give 10 
Riyals to first one and 5 Riyals to second one. 
Cannot choose same person twice. 





Here: 
(1) ORDER MATTERS 
(2) REPETITION NOT ALLOWED 





Possible choices: six 








Question1: Does order matter? 
Question2: Is repetition allowed? 
You are allowed to choose the same person 


twice. 

Here: 

(1) ORDER MATTERS 

(2) REPETITION IS ALLOWED 
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Game: choose two of these. | will give 10 
Riyals to first one and 5 Riyals to second one. 











a a 






Possible choices: nine 








Note: most number of possibilities when order 
matters and repetition is allowed! 





Recall5!=5x4x3x2x1 


How to choose counting formula! And 0! = 


Total objects =n Note: for “without replacement” cases 
Objects to choose (or number of places to fill) =r roan 








Order Repetition 
Allowed? 2 
Use Powers 
n! 
Cp — ——— 
Yes No Use Permutations us (n—r)! 
n! 
No No Use Combinations Ge a 
(n—r)lr! 


EE 302 - Probabilistic Methods in Electrical Engineering / Dr. 


Naveed R. Butt @ Jouf University ee 


Applying the counting formulas 











Here: nC, = o 
Game: choose two of these three (1) ORDER DOESN’T MATTER r (n—r!r! 
students and | will give 10 Riyals to each. (2) REPETITION NOT ALLOWED ai 
Cannot choose same person twice. (3)n =3,r=2 aC, = ! 3 

, (3 — 2)!2! 

Game: choose two of these three Here: np. — n! 
students and | will give 10 Riyals to first (1) ORDER MATTERS " (n-r)! 
one and 5 Riyals to second one. Cannot (2) REPETITION NOT ALLOWED 31 
choose same person twice. (3)n=3,r=2 oP. = (3-2)! = 3!=6 
Game: choose two of these three mete. 
students and | will give 10 Riyals to first (1) ORDER MATTERS ie a @ 


(2) REPETITION ALLOWED 


one and 5 Riyals to second one. You are 
(3)n=3,r=2 


allowed to choose the same person twice. 
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Example Problems 
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Ha 


Questions??? Thoughts? ? 


° 


e 
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Random Event > Random Variable 


e Let’s talk about the next student who enters the room. 
¢ X = number of mobiles he has 
¢ Y=his height 
¢ Z=is he from Jouf region? 


¢ What are the possible values of X, Y, and Z? 


Random Event > Random Variable 


¢ Let’s talk about the next student who enters the room. 
¢ X = number of mobiles he has 
¢ Y =his height 
¢ Z=is he from Jouf region? 

¢ What are the possible values of X, Y, and Z? 
¢ X=0,1, 2,3... mobiles. Only fixed values (discrete) 
¢ Y = Height in cm. Any value in a certain range (continuous) 
¢ Z= Yes or No (just description, no numeric values) 


Random Event > Random Variable 


e All three, X, Y, and Z are random events, but ... 
¢ X is a discrete random variable 
¢ Y is a continuous random variable 


¢ Zis not a random variable! (to qualify for random variable it must have 
numeric values) 


What is a Random Variable? 


¢ A random variable 


1. 


Zz. 
3. 
4 


Is a variable whose value depends on the result of a random occurrence 
Always takes numeric values (NOT descriptions) 

Can be continuous or discrete 

Must satisfy some additional conditions (such as P(X = oo) = 0 etc.) 


Random Event > Random Variable 


¢ We can convert a random event to random variable by assigning 
numeric values to descriptive outcomes 


Z = Next student who enters the room, is he from Jouf region? = Not a random variable yet. 


Possible outcomes 
(descriptive) 


Possible outcomes 
(numeric) 





Z = 1 if next student entering the room is from Jouf = Now Z is a random variable!! 
Z = O if next student entering the room in not from Jouf 
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Random Event > Random Variable 


¢ We can convert a random event to random variable by assigning 
numeric values to descriptive outcomes 


¢ Z = Next student who enters the room, is he from Jouf region? 
¢ Possible outcomes, Z = Yes or No 
¢ Zis not a random variable as its values are not numeric 
¢ We can convert Z into a random variable by mapping Yes = 1, No=0 


¢ Then possible outcomes are Z = 1 or 0 
¢ Now Zis a random variable 


Collecting Probabilities 


¢ Once again, let’s talk about the next student who enters the room. 
¢ X = number of mobiles he has 
e Y=his height 
¢ X and Y are random variables. We will denote the values they take by small 
letters x and y. 
¢ How do we represent the probabilities of the different values of X and 
Y? 


We could make a table ... 


0 0.1 
0.6 
2 0.2 


X = number of mobiles next 
student entering the room has 
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10 


Or, we could draw a graph... 


P(Y = y) 


Y = height of the next student 
entering the room 


y 
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Or, we could write probabilities as a function 
(formula) ... 


Z = result of rolling a die 


X = student height 
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Collecting Probabilities 


¢ How do we represent the probabilities of the different values of X and 
Y? 
¢ We could write them in a table 
¢ Draw them as a graph 
° Or, 





EE 302 - Probabilistic Methods in Electrical Engineering / Dr. Naveed R. Butt @ Jouf University 


Probability Mass Function (PMF): f(x) 


¢ First we will focus on discrete random variables (e.g. X in previous 
slides) 


¢ The formula showing the probabilities of different values of X is called 
probability mass function (PMF) 


f(x) = P(X =x) 


I | Flaeéyiee CAAA ate | PT) ANaywood DP Riitt A) Inisf lL Inivarcity 
ic Methods in Electrical Engineering / Dr. Naveed R. Butt @ Jouf University 





Three Conditions f(x) Must Satisfy 


For a discrete random variable X with possible values x, x, ..., X,, a probability 
mass function is a function such that 


(1) f(x) 20 

(2) > f(x) =1 
i=l 

G) fa) =PX =) 


steaming 
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Let’s play ... 





¢ Let’s assume there is a bag with three balls in it with numbers 1 — 3 
written on them. You draw one ball at random. 


¢ X = number written on the ball (random variable) 
¢ x= 1, 2, 3 (possible values of X) 


+ F(x) = P= 2) ; 


1 





Note that f(x) satisfies the 


three conditions 2 
1. Its always non-negative ==> 
2. Sumsuptol 3 


3. It assigns probabilities to 
each value of X 
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Cumulative Distribution Function (CDF): F(x) 


¢ Sometimes we like to talk about what range of values a random 
variable may take 


¢ What is the probability that the next student entering the room has more 
than two mobiles? i.e., P(X > 2) = ? 


¢ What is the probability that the next student entering the room has two 
mobiles or less?, i.e., P(X < 2) =? 


¢ For such cases, the Cumulative Distribution Function (CDF) is useful. 


Cumulative Distribution Function (CDF): F(x) 


re) = Pos 


F(x) = dixjsx f (i) 








Cumulative Distribution Function (CDF): F(x) 


f(x) 


o.8 
os 
—_ 
= 
3 

> = 06 
= a, 
E 2 
oa 

. Emir 
= 
= 
oS 


- 
Bs 


Scores od 





Dice Side 
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a8 


Cumulative Prohability 


P(X < 3) =sum of all these possibilities = 
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Three Conditions F(x) Must Satisfy 


For a discrete random variable X, F(x) satisfies the following properties. 
(1) F(x) = P(X Sx) = By sx fi) 
(2) 0=F(x)=1 
(3) Ifx<y, then F(x) = Fy 
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Let's play ... 





¢ Let’s write F(x) for the three balls example 


ee 
. i 


Note that F(x) satisfies the 

three conditions 
(1) F(x) = P(X Sx) = Zyxx f(%) 
(2) 0 = Hx) =1 
(3) Ifxsy, then F(x) = Fy) 
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We can use F(x) to find many kinds of 
probabilities (ranges and values) 


@ P(X< a)= F(a) 
P(X= a)=P( Xs a) — P (X< a) 
=P(X< a)— P(X< a-—1) 
=1-— F(a) — F (a-1) 
P(X>a)=1-P(Xs a)=1 — F(a) 
P(X> a)=1-— P(X<a)=1- P(X¥ a -1)= 1- F(a-1) 
P(X<a)=P(XK a — 1)=F (a -1) 
P(a <X < b)= F(b) — F(a) 
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Test the relations on previous slides on this 
example 


= 
= 
S 
= 
° 
- 
a 





f(x) 
F(x) 


0.8 
o fi 


Oa. 


Cumulative Prohability 


Od 


Scores 





Dice Side 
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Joint Probability Distribution (JPD): f(x, y) 


¢ Sometimes we are interested in finding the probability of two things 
happening at the same time 
¢ What is the probability that next student entering the room has two mobiles 
and no pen? 
¢ X =number of mobiles he has 
¢ Y=number of pens he has 
¢ P(X=2 and Y=0)=? 


¢ For such cases, we use the Joint Probability Distribution (JPD) 


Joint Probability Distribution (JPD): f(x, y) 


f(xy) = P(X =x,Y =y) 





Three Conditions f(x, y) Must Satisfy 


The function f(.c,y) is a joint probability distribution or probability mass 
function of the discrete random variables X and Y if 


1. f(x,y) > O for all (x,y), 


2. Ov s(a,y) =1, 
cy 


ew FA Sey Sy) = Fie). 
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Let’s play ... 


¢ Let’s assume there is a bag with three balls in it with numbers 1 — 3 
written on them. You draw two balls (one by one) at random. 
¢ X = number written on the first ball (random variable) 
¢ Y =number written on the second ball (random variable) 
¢ x= 1, 2, 3 (possible values of X) 
e y=1, 2, 3 (possible values of Y) 
g(x) = P(X =x) 
“h(y) = P(Y =y) 
f(y) = PX =x,Y =y) 
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All the possible outcomes (note that 
they are equally likely) 


S = { (1,2) (1,3) (2,1) (2,3) (3,1) (3,3) } <——— 





Note that f(x, y) satisfies the three conditions 
1. f(x,y) > 0 for all (x, y), 
2. > f(x,y) = 1, 
rey 
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f(x) 








Plot of a single random variable distribution is a two Plot of a joint distribution is a three dimensional curve! 
dimensional curve 
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Probability over an areaA 





y<3 


A={x>1y <3} P[(X,Y) € A] = f(2,1) + f(2,2) + f(3,1) + f(3,2) = ; 
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Marginal Distribution 


¢ Sometimes we are interested in questions such as “given the joint 
distribution of X and Y, find the distribution of X” 
¢ This may be the case, for instance, if we have easy access to the joint 
distribution. 
¢ The distribution of X extracted from the joint distribution of X and Y 
(by summing over all possible values of Y) is called the Marginal 
Distribution of X 






Marginal Distribution 


Sum f (x, y) over all 
the possible values of Y 


Sum f(x,y) over all 
the possible values of X 
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g@) = fG1) + fG.2) + £3) _ 

g(x) = marginal 
g(2) = f (2,1) + f (2,2) + f(2,3) distribution of x 
g(3) = FGA) + FBZ) +FG,3) 





A(1) = fC) + f@,1) + F(Z,3) 
h(y) = marginal 
MZ =f) e742) distribution of y 


h@) = fG3) + f(2,3) + f(3,3) 
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Conditional Distribution (CD): f(x | y) 


¢ Sometimes we are interested in finding the probability of an event 
given that another event has taken place. 
¢ What is the probability that a student will get GPA 5 in this semester given 
that his CGPA till previous semester is 4.5? 
¢ X =GPA student will get in this semester 
°* Y= His CGPA now 
¢ P(X =5 given Y = 4.5) =? 


¢ For such cases, we use the Conditional Distribution (CD) 


Conditional Distribution (CD): f(x | y) 


Let X and Y be two random variables. discrete or continuous. The conditional 

distribution of the random variable Y given that X = x is 

f(x,y) 
g(a) 


fly|z) = 


, provided g(x) > 0. 
Similarly, the conditional distribution of X given that Y = y is 


wy) provided h(y) > 0. 





Note: 
g(x) = marginal distribution of X 
h(y) = marginal distribution of Y 
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All the possible outcomes (note that 
they are equally likely) 


S = { (1,2) (1,3) (2,1) (2,3) (3,1) (3,3) } <——— 
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Statistical Independence 


¢ Two random variables are statistically independent if knowledge of 
one does not change the probability distribution of the other. 


How to check statistical independence? 


¢ We can prove that two random variables are statistically independent 
by showing any of the following to be true for all values of x and y. 


f(xly) = 9g) 
fly|x) = hv) 
{© y) = g(x)h(y) Note: 


g(x) = marginal distribution of X 
h(y) = marginal distribution of Y 


All the possible outcomes (note that 
they are equally likely) 


S = { (1,2) (1,3) (2,1) (2,3) (3,1) (3,3) } <——— 





X and Y are not statistically independent, since for many values of x and y 


fay) # g@)h(y) 


eg. f(2,1) # g(2)h(1) 
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Questions??? Thoughts? ? 


° 


e 
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We have previously talked about... 
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Collecting Probabilities 


¢ Next student who enters the room. 
¢ X = number of mobiles he has 
¢ Y =his height 
¢ How do we represent the probabilities of the different values of X 
and Y? 


We could make a table ... 


0 0.1 
0.6 
2 0.2 


X = number of mobiles next 
student entering the room has 
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Or, we could draw a graph... 


P(Y = y) 


Y = height of the next student 
entering the room 


y 
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Or, we could write probabilities as a function 
(formula) ... 


Z = result of rolling a die 


X = student height 
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Collecting Probabilities 


¢ So, we collect probabilities in terms of a probability mass function 
(discrete case) or probability density function (continuous case) 


Number of students 


_@ 20-00 eee fe 9-0 _- 


Student Marks 
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Collecting Probabilities 


Number of students 


Student Marks 
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Collecting Probabilities 


Number of students 


SS Le 


Student Marks 
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Collecting Probabilities 


Number of students 


oO OO | 


Student Marks 
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Two Important Measures of a Random 
Variable: Mean and Variance 


¢ Next we will talk about two important measures of a random variable 
that are often used in analysis and design instead of the full 
probability functions. 


¢ These are Mean and Variance 
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Many names: Mean/Average/Expectation 


Mean = Center of mass for the distribution (balancing point) 


Number of students 





Student Marks 
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Many names: Mean/Average/Expectation 





Mean = Center of mass for the distribution (balancing point) 


Number of students 


Student Marks 
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Many names: Mean/Average/Expectation 





Mean = Center of mass for the distribution (balancing point) 


Number of students 


— ei @° 0a 0 


Student Marks 


EE 302 - Probabilistic Methods in Electrical Engineering / Dr. Naveed R. Butt @ Jouf University 


Many names: Mean/Average/Expectation 


Mean = Center of mass for the distribution (balancing point) 


Number of students 





pe ee 


Student Marks 
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Variance = degree of spread (how much 
variation is there in the data/outcomes?) 


Medium variance 


ERD 


High variance 


<Zhoccee Sih 
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Low variance 





How shall we write Mean and Variance 
mathematically? 


Mean: Let’s play a game... 


¢ Suppose | invite you to a game of toss with following rules 
¢ We toss several times (e.g., 100 tosses) 
e Every time H appears | give you 10 Riyals 
e Every time T appears | take from you 10 Riyals 


¢ How much can you expect to win on average? 
e Is there any point in playing this game? 


Mean: Let’s play a game... 


1 Logicall 
P(H) = P(T) => a 


J J 1 You win 0 Riyals on average 
f(10) = f(-10) = 5 


Mathematically 


E|X] = x, f (x1) + x2f (x2) = 10 xo+ (—10) x55 0 





it 
fe) 5 
Graphically 
A Mean = where should you place the wedge so that the 
-10 5 10 “scales” are balanced 
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Mean: Let’s play a game... 


¢ Suppose now | change the rules slightly 
¢ We toss several times (e.g., 100 tosses) 
e Every time H appears | give you 10 Riyals 
e Every time T appears | take from you 50 Riyals 


¢ Now how much can you expect to win on average? 
¢ Should you play such a game? 


IaNnN97 Drahsahilictic Mathadc in Clartric- Crnacinaaring /P Alayvroon PR tte TH) Init | Inivarcit 
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N] eR 


f(x) 


On average, you stand to 


Mean: Let’s play a PamMe a lose 20 Riyals per toss 


1 
P(H) = P(T) = 3 Logically 


ff , 


f(10) = f(—50) == You will expect to lose more money 
2 


Mathematically 


E|X] = x, f (x1) + x2f (x2) = 10 xo+ (—50) x55 —20 





Graphically 
| , Mean = where should you place the wedge so that the 
ao x 10 “scales” are balanced 


EE 302 - Probabilistic Methods in Electrical Engineering / Dr. Naveed R. Butt @ Jouf University 


Mean: Let’s play a game... 


¢ Suppose we change the game to rolling a die 
¢ We roll several times (e.g., 100 rolls) 
e Every time a number above 2 appears | give you 10 Riyals 
e Every time a number less or equal to 2 appears | take from you 10 Riyals 


¢ Now how much can you expect to win on average? 
¢ Should you play such a game? 


Mean: Let’s play a game... 


2 1 
_ 2 P(X <2)= ; = A Logically 


PXS2Z)= 


| 


J 2 4 1 You expect to win more 
f (10) = a f(—10) = 3 


Mathematically 


E|X] = x,f (x1) + x2f (x2) = 10 xS+ (—10) xo 3.33 


Wi] rN 


f(x) 


Wl eR 


Graphically 





x Mean = where should you place the wedge so that the 
-10 ; 10 “scales” are balanced 
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SO, 


The mean or expected value of a discrete random variable xX, 
denoted lly or E(X), is given by 


by = E(X) =) xf) 


x 
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¢ Let’s assume there is a bag with three balls in it with numbers 1, 3, 
and 5 written on them. You draw one ball at random. 
¢ X = number written on the ball (random variable) 
° x= 1, 3,5 (possible values of X) 


f(x) = P(X = x) 


nro) = Dare) (1x5) +(245) +(545) = 


:  F 
: ei 


5 
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¢ Let’s assume there is a bag with three balls in it with numbers 1, 3, 
and 5 written on them. You draw one ball at random. 
¢ X = number written on the ball (random variable) 
¢ x= 1, 3, 5 (possible values of X) 


f(x) = P(X = x) 


f (x) ux = E00 = Y afta) = (13) + (3x5) +(5x5)=3 


x 
+ + 
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Variance 


The variance of a discrete random variable X, denoted o¥ or 
V(X), is given by 


Ox = V(X) = E(X —- px) 
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Several equivalent formulations of variance 


ae hee 
B= S003 


x 
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¢ Let’s assume there is a bag with three balls in it with numbers 1, 3, 
and 5 written on them. You draw one ball at random. 


¢ X = number written on the ball (random variable) 
° x= 1, 3,5 (possible values of X) 


f(x) = P(X = x) 





entpeadwne-fishisfenleteddes 


ee ee FOO 
3 [=o 6? = ¥, x2 F(x) — 2 = (ie *) + ((3)?x = }( (5)? :) ee 


5 
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Standard Deviation 


The square root of variance is called standard deviation, and denoted by oy 


Oy = J/V(X) 
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¢ Let’s assume there is a bag with three balls in it with numbers 1, 3, 
and 5 written on them. You draw one ball at random. 
¢ X = number written on the ball (random variable) 
° x= 1, 3,5 (possible values of X) 
° f(x) = P(X =x) 


f (x) 





_ _ _ 1 1 1 _ 
ux = E00 = Y xf = (15) +(3 <5) +(5 «3)=3 


1 : ¥ 
3 of = Dex? f(x) — wh = ((1)2x 5) + ( (8)? 3)+( (6)? 3) - @)? = 2.66 
5 


Ox = V2.66 = 1.63 
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What is the mean of a constant (or a non-random variable) ? 
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What is the mean of a constant (or a non-random variable)? 


If C is a constant, then 
A constant takes only one 
E(C) — C e.g.£(10) =10 <————————.§ value, so the “balancing 
point” is that value itself 
If Z; is anon-random variable, then 


| 
E (Z,) — Lt e.g. if Z, = t, then we already | A 
know that Z; = 1,Z, = 2,... (i.e., 0 


10 





the values of Z; are non-random) 
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What is the variance of a constant (or a non-random variable)? 
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What is the variance of a constant (or a non-random variable)? 


If C is a constant, then 
V(C) — 0 e.g. V(10) = 0 A constant takes only one ; 
value, so there is no “spread 
If Z; is anon-random variable, then 


| 
V(Z;) — 0 e.g. if Z, = t, then we already | A 
know that Z; = 1,Z, = 2,... (i.e., 0 


10 





the values of Z; are non-random), 
and so we know that V(Z,) = 
V(1) = 0,V(Z,) = V(2) = O,... 
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Some important properties of the mean 


If X is arandom variable and a and b are constants, then 


E(aX +b) = aE(X) +b 


For any two random variables X and Y, we have 


E(X +Y) = E(X) + E(Y) 


E(X —Y) = E(X)-E(’) 





For two independent random variables X and Y, we have 


E(XY) = E(X)E(Y) 
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Some important properties of the mean 


If X is arandom variable and v(X) is some function of X, then 


E@(X)) = ) vf @) 





x 


For example, if v(X) = X°, then 


E((X)) = E(X*) = ) 28 F@) 





x 
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Some important properties of variance 


If X and Y are any two random variables and a, b are constants, then 


V(axX + bY) = a*V(X) + b*V(Y) + 2abE((X — wy) (Y — py)) 


If X and Y are two independent random variables and a, b are constants, then 
V(aX + bY) = a*V(X) + b*V(Y) 
V(aX — bY) = a2V(X) + b?V(Y) 
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Example 1 


Consider two zero-mean unit variance random variables X, Y. Define a new random variable W = 2X+yY. 
Given that X and Y are independent, find mean and variance of W. 


We are given that 


Lx = My =0 
6, = 07 =1 
And 
X,Y are independent 
Therefore 


Uy = E(W) = E(2X + Y) = 2E(X) + E(Y) =0 
V(W) =V(2X + Y) = 4V(X) + V(Y) = 441=5 
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Example 2 


Consider two zero-mean unit variance random variables X, Z. Define a new random variable W = 2X+2Z. 
Given that E(XZ) = 1, find mean and variance of W. 


We are given that 


lx =Uz =0 
67 =07 = 1 


Therefore 
Uy = E(W) = E(2X + Z) = 2E(X) + E(Z) = 0 
V(W) =V(2X + Z) = 4V(X) + V(Z) + 2(2)E((K — wy) (Z— wz) = 44+14+4(114+04+040) =9 
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Questions??? Thoughts? ? 


° 


e 
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We have previously talked about... 


EE 302 - Probabilistic Methods in Electrical Engineering / Dr. Naveed R. Butt @ Jouf University 


Probability Mass Function (PMF): f(x) 


jo) =cr=*) mmm 


For a discrete random variable X with possible values x), x, ..., X,, a probability 
mass function 1s a function such that 
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What are some of the common distribution 
functions for discrete random variables? 


¢ Next, we will look at some probability distributions (f(x)), that are 
often used in practice for discrete random variables 


¢ These include 
¢ Uniform Distribution 


¢ Binomial Distribution 
¢ Geometric Distribution Based on Bernoulli Trials 


¢ Negative Binomial Distribution 
¢ Poisson Distribution 


Discrete Uniform Distribution = when all 
possible outcomes have equal probabilities 
¢ X= result of a rolling a die 
x = 1,2,3,4,5, 6 (possible values of X) 
¢ All outcomes have equal probability 

1 
‘Fes a X= 1,2,3,4,5,6 


0, otherwise 
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Discrete Uniform Distribution = when all 
possible outcomes have equal probabilities 


A random variable X has a discrete uniform distribution if it has a finite 
set of outcomes (Say x1, X2,X3, ...,X,), and each outcome has exactly 
the same probability. The distribution of X with n equally probably 
outcomes may then be written as 


~~ 


x= X4,X2,X3, iy Xn 


f(x) = 


Os lr 


otherwise 


~ 
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Discrete Uniform Distribution = when all 
possible outcomes have equal probabilities 


Mean and Variance of a uniformly distributed random variable X 


1 
f(x) = a’ X= X45 X71 XZ,y wey Xn 
0, otherwise 


E[X] = py = D7) = =" 


t=1 


1 n 
VIX] =) = ax)?f@) ==) Gi - Hx)? 


i=1 
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Discrete Uniform Distribution 


Special case: when x; are integers in range [a b| 


Suppose XY is a discrete uniform random variable on the consecutive integers 
a,a+1,a+2,..., 6, fora = b. The mean of X 1s 


b+a 
2 





p = E(X) = 


The variance of Y is 
(oe 1 1 
12 


<a 
o = 
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Example 


1 
i = 1,2,3,4 

If ToS 6’ x , feo ,9,6 
0 


otherwise 


~ 


then 





6+1 
E[X] = —— =3.5 
2 
(6-1+1)?-1 35 
VIX] = 5 = 75 = 2.92 
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What are some of the common distribution 
functions for discrete random variables? 


¢ Next, we will look at some probability distributions (f(x)), that are 
often used in practice for discrete random variables 


¢ These include 
¢ Uniform Distribution 


¢ Binomial Distribution 
¢ Geometric Distribution Based on Bernoulli Trials 


~ 


¢ Negative Binomial Distribution 
¢ Poisson Distribution 
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LLL 


LLR 
LRL 


LRR 
RLL 


RLR 
RRL 


RRR 


11 





p 
Note that: q _ -- eo R 
r- © 


Same experiment is repeated several times 


p L 
- Each time there are only two possible outcomes a ae R ae 


(L or R) 


- Each time the probabilities of the two outcomes a= R 
are the same (P(L) = p, P(R)= q) 
All the repetitions are independent of each other 
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LLL 


LLR 
LRL 


LRR 
RLL 


RLR 
RRL 


RRR 
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Bernoulli Process & Bernoulli Trials 


¢ An experiment that satisfies the conditions we saw in the previous 
example is called a Bernoulli Process, and each repetition is called a 
Bernoulli Trial. 


In summary, a Bernoulli Process is a random process 
that satisfies the following conditions 
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Some Questions we could ask 


¢ What is the probability that in the n repetitions you take exactly x left 
turns? 


¢ What is the probability that the first left turn you take is at repetition 
number x? 


¢ What is the probability that it takes you x repetitions to take exactly r 
left turns? 


Each of these leads to a different random variable 
x, with its own probability mass function f (x) 


What is the probability that you take three left turns? 


Probability (first left and second left and third left) = ? 


a 

Let t; = first turn, t2 = second turn, t3 = third turn p L & 
ee a R 

Then, L & L 


P(t; =LNt> =LNtz =L) =? p ee, 


= P(t, = L)P(t, = L) P(t = L) 


=pxpxp=—p° & 





—— 
q 
p L 
Recall that for independent events A, B, and C q = 
we have P(AN BNC) = P(A)P(B)P(C) R —— 
q 
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LLL 


LLR 
LRL 


LRR 
RLL 


RLR 
RRL 


RRR 


What is the probability that you take exactly two left turns? 


P(t, =LNt, =LNtz =R) 


pL 

+P(t, =RNt, =LNtz =L) p 
_~- ae 
= ppq + pqp + app = 3p"q & L 


N 


p a ae 


6 
. 


i “ 


A 


eS 


Next, We will use this understanding to eee 
answer several such questions, and R 


develop various probability distributions 
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LLL 


LLR 
LRL 


LRR 
RLL 


RLR 
RRL 


RRR 


What are some of the common distribution 
functions for discrete random variables? 


¢ Next, we will look at some probability distributions (f(x)), that are 
often used in practice for discrete random variables 


¢ These include 
¢ Uniform Distribution 


¢ Binomial Distribution 
¢ Geometric Distribution Based on Bernoulli Trials 


e Negative Binomial Distribution 
¢ Poisson Distribution 
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Binomial Distribution 


Consider an experiment consisting of n Bernoulli trials (i.e., independent trials 





with constant probability of “success” denoted p) 


Then the probability distribution of X is given by 


eo pt pe ee ie 
7 


And we say that X has a Binomial Distribution 
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Defining the setup 
(experiment) 


Defining the random 
variable 


Finding formula for 
distribution of x 


Naming the 
distribution 
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Geometric Distribution 


Consider an experiment consisting of Bernoulli trials (i.e., independent trials with 


constant probability of “success” denoted p) 





Then the probability distribution of X is given by 


fx) =(1—-pyf px =1,2,... 


And we say that X has a Geometric Distribution 
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Defining the setup 
(experiment) 


Defining the random 
variable 


Finding formula for 
distribution of x 


Naming the 
distribution 
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Negative Binomial Distribution 


Consider an experiment consisting of Bernoulli trials (i.e., independent trials with 


constant probability of “success” denoted p) 





Then the probability distribution of X is given by 


X—- 1] ae 
fe) =("— Jape rane thr t2... 
: 





And we say that X has a Negative Binomial Distribution 
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Defining the setup 
(experiment) 


Defining the random 
variable 


Finding formula for 
distribution of x 


Naming the 
distribution 


20 


Recall ... 


n n! 
"Cor = —_———- 
r} (n-r)!r! 


Recall5!=5x4x3x2x1 
AndO!=1 
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Interpretations ... 


Total number of trials =n 
x successes Failures=n—x 


\ 7 


fo) a : p “(1 - a x=0, 1.207 Binomial Distribution 


~ 


P(success) = p P(failure) =1-p 


Number of ways we can have x 
successes in 7 trials knowing 
that the order does not matter 
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What is the probability of two success in three trials? 
P(SONSNF)+P(SNFNS)+PFASNS) 


= pp(1—p)+p(1—p)p+(—p)pp = 3p*(1 — p) 


Using the Binomial Distribution formula 
with n = 3,x = 2 we get the same result 


se) = (") p* = py << et 


3! - 
pa=(; )p’ (1—p)= Baar? 1 —P) = 3p — p) we 
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SSS 


Soh 
SiS 


SFF 
FSS 


FSF 
FFS 


FFF 


Interpretations ... 


P(failure) =1-p 


! 


7 (x) =S (1 = py. ‘Dp y= Ie 2, eee Geometric Distribution 


P(success) = p 


There has to be at least one trial for 
one success to occur (so no x = 0) 
There are a total of x — 1 failures 
Success occurs in trial number x but before the first success occurs 
not before 
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What is the probability of that first success occurs in third trial? 
P(FOFNS) 
=(1—p)A—p)p = (1—p)’p 


Using the Geometric Distribution formula 
with x = 3 we get the same result 


f(x) = (1 - py 'p pF 


f(3)=(-p)*p 
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SSS 


SSF 
SFS 


SFF 
FSS 


FSF 
FFS 


FFF 


Interpretations 


x — r failures 


X—- | 
= (I 


Ja = pyre 


P(failure) =1-p 


By definition, we know that the last trial is a 
success, thus we only have to see in how many 
ways can we have remaining r — 1 successes in 
the remaining x — 1 trials, knowing that order 
does not matter in these x — 1 trials. 
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men 


There have to be at least r trials 


for r successes to occur 
r successes 


Negative Binomial 
Distribution 


SF LP oe Zo. 


‘“ 


P(success) = p 
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What is the probability that it takes three trials (not more or less) to get first two successes? 


P(SQNFOS)+P(FOSNS) 


a 
= p(1—p)p+(1—p)pp = 2p*(1—p) S 
S S 
p 
p . - a 
Using the Negative Binomial Distribution 1-p E 
formula with x = 3,r = 2 we get the 
same result P ° 
S 
p 
p— | 1-p ie 
fix) =(" | J pyr" 
‘ Vr 28 | . Bn a S 
1-p F 
(3y=(>— + \pta—p) = 20 -p) 1p 
PB=\5_4)P PI =n? P 
= 2p*(1—p) 
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SSS 


SSF 
SiS 


SFF 
FSS 


FSF 
FFS 


FFF 


Summary of Properties (Range, Mean, Variance) 


Binomial Distribution 


10) =(") px ny 
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Geometric Distribution 
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What are some of the common distribution 
functions for discrete random variables? 


¢ Uniform Distribution 

¢ Binomial Distribution 

¢ Geometric Distribution Based on Bernoulli Trials 
¢ Negative Binomial Distribution 

¢ Poisson Distribution 
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What are some of the common distribution 
functions for discrete random variables? 


¢ Uniform Distribution 

¢ Binomial Distribution 

¢ Geometric Distribution Based on Bernoulli Trials 
¢ Negative Binomial Distribution 


¢ Poisson Distribution 


Based on a special “counting” 
process called Poisson Process 
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A Counting Process 


¢ We often need to count random events, e.g., 
¢ How many customers visit a shop in a given time duration 
¢ How many of my students fall asleep during my lecture 
¢ How many people visit a given website in a given time duration 
¢ How many calls you get during the lecture 
¢ How many drops of rain fall in a given area 


A Counting Process 


¢ Let X be the number of some specific random events that occur in 
duration, say t ( t can be any unit of time, length, area, volume etc.) 
e E.g., if t = 2 hours, then X = {number of calls you get in two hours} 


e Assume that the counting begins with zero, i.e., X = 0 fort = 0 


8am 9am 10am 11am 


a | ' 


Wie ear calle mone Hour X = # of calls in two hours 


| 
! 


X = # of calls in three hours 
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Poisson Process 


¢ Let X be the number of some specific random events that occur ina 
duration, say t 


¢ We say that X is a Poisson Random Variable if it satisfies the 
conditions of a Poisson Process, which are as follows 
¢ The number of events in two non-overlapping intervals are independent 


¢ The number of events during an interval depend only on the length of the 
interval 


¢ The probability of two events occurring at exactly the same time is zero 


¢ The average number of events per unit interval (time, distance etc.) is 
constant (usually denoted by A) 
¢ Thus average number of events in interval t is At 


Poisson Distribution 


The probability distribution of the Poisson random variable X, representing 
the number of outcomes occurring in a given time interval or specified region 
denoted by f, is 


e “ier 


p(x; At) = - 


| 

Li: 

where A is the average number of outcomes per unit time, distance, area, or 
volume and e = 2.71828.... 
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Examples 
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Questions??? Thoughts? ? 


° 


e 
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Continuous Random Variables and Probability 


Distributions 





3-1 CONTINUOUS RANDOM VARIABLES 


A continuous random variable has an infinite number of possible values & the 
probability of any one particular value is zero. A continuous random variable is a 
random variable that can assume any value in some interval of numbers, and are thus 
NOT countable. 
Examples: 

Vv The time that a train arrives at a specified stop 

v The lifetime of a transistor 

v A randomly selected number between 0 and 1 

VY Let R be a future value of a weekly ratio of closing prices for IBM stock 

VY Let W be the exact weight of a randomly selected student 


3-2 PROBABILITY DISTRIBUTIONS AND PROBABILITY DENSITY 
FUNCTIONS 
A random variable is said to be continuous if there is a function f (x) with the 
following properties: 

a) Domain: all real numbers 

b) Range: f (x) = 0 

c) The area under the entire curve is 1 
Such a function f (x) is called the probability density function (abbreviated p.d.f.) 
The fact that the total area under the curve f (x) is | for all X values of the random 
variable tells us that all probabilities are expressed in terms of the area under the curve 
of this function. 





The function f(r) is a probability density function (pdf) for the continuous 
random variable X, defined over the set of real numbers, if 


1. f(x) > 0, for all x € R. 


2 | oe Fin ee = 1. 





3. Pa<X <b)=f- f(a) de. 


The important point is that f(x) is used to calculate an area that represents the probabil- 
ity that XY assumes a value in [a, 5]. For the current measurement example, the probability that 





If Xis a continuous random variable, for any x, and x, 


Ay sXS%) = AY <XS%)= AY = XK< wy) = AY <X< 4%) (42) 
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3-3 CUMULATIVE DISTRIBUTION FUNCTIONS 
The cumulative distribution function F(.c) of a continuous random variable 
X with density function f(r) is 





Pie] PFs = 4)= [ f(t) dt, for —o<a£< oo. 





@ 0 < F(x) <I, for all x 

Notice that in the definition of F(x), any < can be changed to = and vice versa. That is, 
F(X) can be defined as either 0.05x or 0 at the end-point x = 0, and F(X) can be defined as 
either 0.05x or 1 at the end-point x = 20. In other words, F(X) is a continuous function. For a 
discrete random variable, F(X) is not a continuous function. Sometimes, a continuous random 
variable is defined as one that has a continuous cumulative distribution function. 

The probability density function of a continuous random variable can be determined from 
the cumulative distribution function by differentiating. Recall that the fundamental theorem of 
calculus states that 


x 


d 
= u) du = f(x 
& | Aa) du = 109 
As an immediate consequence of Definition 3.7, one can write the two results 
dF (x) 


P(a< X <b) = F(b) — F(a) and f(z) = — 





if the derivative exists. 

3-4 MEAN AND VARIANCE OF A CONTINUOUS RANDOM VARIABLE 
Let X be a random variable with probability distribution f(a). The mean, or 
expected value, of X is 


w= E(X) =) af(z) 


ac 


if X is discrete, and 


p= B(4) = [- uf(a) dx 





if X is continuous. 
Let X be a random variable with probability distribution f() and mean jp. The 
variance of X is 


o*? = E[(X —p)*] = Soe — p)* f(z), if X is discrete, and 


x 


o* = E[(X — »)*] = / (x — ps)? f(a) dex, if X is continuous. 







The positive square root of the variance, o, is called the standard deviation of 
X. 
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The variance of a random variable X is 


= E(X?) — p’. 





3.5 TWO OR MORE RANDOM VARIABLES 
Joint Probability Distributions 


The function f(x,y) is a joint density function of the continuous random 
variables X and Y if 


1. f(x,y) => 0, for all (x, y), 


2. | 3 f(x,y) dx dy = 1, 
3. P[(X,Y) € A]=f J, f(x,y) dx dy, for any region A in the «ry plane. 





Given the joint probability distribution f(r, y) of the discrete random variables 
X and Y, the probability distribution g(a) of X alone is obtained by summing 
f(x,y) over the values of Y. Similarly, the probability distribution h(y) of Y alone 
is obtained by summing f(x,y) over the values of X. We define g(a) and h(y) to 
be the marginal distributions of X and Y, respectively. When X and Y are 
continuous random variables, summations are replaced by integrals. We can now 
make the following general definition. 





The marginal distributions of X alone and of Y alone are 
=So f(x,y) and h(y)= 0 f(xy) 
y z 


for the discrete case, and 


g(e)= f fley)dy and hy) =f fey) ae 





for the continuous case. 





If we write the marginal fx (a) in terms of the joint density, then this becomes 


E[X] = rp i teceddedy 
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Let X and Y be two random variables, discrete or continuous. The conditional 
distribution of the random variable Y given that X =r is 
f(y|z) = La, y) provided g(x) > 0. 
g(x) 
Similarly, the conditional distribution of X given that Y = y is 
ae . 
f(zly) = i y) provided h(y) > 0. 
h(y) 





b 
Pia<X<b|Y=y) =| f(aly) dx. 


Definition 4. Let X,Y be random variables (discrete or continuous). 
joint (cumulative) distribution function is 


Fx y (a, y) = P(X xf, r < y) 
If X and Y are jointly continuous then we can compute the joint cdf from 


their joint pdf: 
© y 
Fy y(g, y) = / / flu, v) tw du 


If we know the joint cdf, then we can compute the joint pdf by taking partial 
derivatives of the above : 
fad 


OxOu 





Fy y(x,y) = f(@.y) 


Statistical Independence 
It should make sense to the reader that if f(x|y) does not depend on y, then of 
course the outcome of the random variable Y has no impact on the outcome of the 
random variable X. In other words, we say that X and Y are independent random 
variables. We now offer the following formal definition of statistical independence. 


Let X and Y be two random variables, discrete or continuous, with joint proba- 
bility distribution f(«,y) and marginal distributions g(a) and h(y), respectively. 
The random variables X and Y are said to be statistically independent if and 
only if 


f(x,y) = g(x)h(y) 


for all (x, y) within their range. 
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Let X and Y be random variables with joint probability distribution f(x,y). The 
covariance of X and Y is 


Oxy = E[(X - px)(¥ - wy)) = >> o(@ - ux) (y — by) f(a) 


¥ 


if X and Y are discrete, and 
oxy = BUX —ax¥—wr= ff nx) ~ sy) Flea) de dy 


if X and Y are continuous. 











The covariance of two random variables X and Y with means jy and jy, respec- 
tively, is given by 


Oxy = E(XY) — pxpy. 





BIXY] = ff ey fx(0) fr(y) dedy 


Theorem 2. Jf X and Y are independent and jointly continuous, then 


E[XY] = E[X]E[Y] 
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Continuous Random Variables and Probability 





Distributions 


3-6 CONTINUOUS UNIFORM DISTRIBUTION 

One of the simplest continuous distributions in all of statistics is the continuous 
uniform distribution. This distribution is characterized by a density function 
that is “flat,” and thus the probability is uniform in a closed interval, say [A, B]. 
Although applications of the continuous uniform distribution are not as abundant 
as those for other distributions discussed in this chapter, it is appropriate for the 
novice to begin this introduction to continuous distributions with the uniform 
distribution. 





Uniform The density function of the continuous uniform random variable X on the in- 
Distribution — terval [A, B] is 


1 

f(a; A, B) = B=A% A<a<B, 

: 0, elsewhere. 
The density function forms a rectangle with base B—A and constant height git: 
As a result, the uniform distribution is often called the rectangular distribution. 
Note, however, that the interval may not always be closed: |A, B]. It can be (A, B) 
as well. The density function for a uniform random variable on the interval [1, 3} 
is shown in Figure 6.1. 


1 
b-a 
f(x)=0 otherwise 





f(x) = 


if a<x<b fe) 





~ 
| 
a 


a b x 


Figure 4-8 Continuous uniform 
probability density function. 





The mean and variance of the uniform distribution are 








18.4 2 G29 
L= a and o = 7 
Proof 
2 
Xx _ 05¥ |? (at d) 





Bat | 2 b-a 





a a 


The variance of X is 





(CY SN] oa 


= b—a 3(b— a) a 12 
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The cumulative distribution function of a continuous uniform random variable is ob- 
tained by integration. If a< x < b, 


F(x) = mc — a)du = x/(b— a) — a/(b— a) 


Therefore, the complete description of the cumulative distribution function of a continuous 
uniform random variable is 


0 X<a 
F(x) = ((x- a)\/(b- a) asx<b 
1 bez 


3-7 NORMAL DISTRIBUTION 


Undoubtedly, the most widely used model for the distribution of a random variable is a 
normal distribution. Whenever a random experiment is replicated, the random variable that 
equals the average (or total) result over the replicates tends to have a normal distribution as 
the number of replicates becomes large. De Moivre presented this fundamental result, known 
as the central limit theorem, in 1733. Unfortunately, his work was lost for some time, and 
Gauss independently developed a normal distribution nearly 100 years later. Although De 
Moivre was later credited with the derivation, a normal distribution is also referred to as a 


Gaussian distribution. 
The theoretical basis of a normal distribution is mentioned to justify the somewhat com- 


plex form of the probability density function. Our objective now is to calculate probabilities 
for a normal random variable. The central limit theorem will be stated more carefully later. 

Random variables with different means and variances can be modeled by normal proba- 
bility density functions with appropriate choices of the center and width of the curve. The 
value of A(X) = p determines the center of the probability density function and the value of 
YX) = o° determines the width. Figure 4-10 illustrates several normal probability density 
functions with selected values of x and o*. Each has the characteristic symmetric bell-shaped 
curve, but the centers and dispersions differ. The following definition provides the formula for 
normal probability density functions. 


f(x) o2=1 





o2=1 


o2=4 


=5 ji= 1S x 


Figure 4-10 Normal probability density functions for 
selected values of the parameters x and 0”. 
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Normal 
Distribution A random variable X with probability density function 


—(«— py? 


f(x) = et —0 << ¥< 0 (4-8) 


1 
V 210 


is a normal random variable with parameters p, where —%© < pp < %, ando > 0. 
Also, 


AUX) =e and. V(X) = 07 (4-9) 


and the notation Mw, 0”) is used to denote the distribution. 





Proof: To evaluate the mean, we first calculate 


E(X —p) =| TH 42") ae. 


coo V 270 





Setting z = (a — y)/o and dx =o dz, we obtain 


E(X — yp) = al ze 2” dz =0, 


since the integrand above is an odd function of z. Using Theorem 4.5 on page 128, 
we conclude that 


BUA) =p 
The variance of the normal distribution is given by 


E((X — p)?] = == | (a — p)2e7 3l(e—#)/o1° dx. 


210 JS — se 
Again setting z = (1 —y)/o and dx =o dz, we obtain 
E((x — 1)?) = = . fe dz. 
Vir Ie 
Integrating by parts with u = z and dv = ze-* /? dz so that du = dz and v = 
—e7?"/2, we find that 


2 oo oo 
E|(X — p)?] = Te [=e igs +f e7? /2 i) =o°(+1) =o°. 





J 


Some useful results concerning a normal distribution are summarized below and in 
Fig. 4-12. For any normal random variable, 


Pw -a< X<pto) = 0.6827 
Pp — 20 < X< p + 20) = 0.9545 
Pw — 30 < X< p + 30) = 0.9973 


Also, from the symmetry of f(x), AX > pw) = AX < p) = 0.5. Because x) is positive for 
all x, this model assigns some probability to each interval of the real line. However, the 
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fo) 


fo) 





p-30 w-20 w-o bh uta pt+2a0 wt+30 x 
|<— 68% —>| 
i. 95% ——_—_—_>| 
10 13 x ———— 99.7% > 
Figure 4-11 Probability that X > 13 for a normal 


Figure 4-12 Probabilities associated with a normal 
random variable with p = 10 and o? = 4. 


distribution. 


The Gaussian random variable is described in terms of two parameters m € R and 
ao > Oby the PDF 


1 =s (x-m)? 
Pam 20? (2.3-8) 


We usually use the shorthand form \/(m, 7) to denote the PDF of Gaussian random 
variables and write X ~ MN (m, o”). For this random variable 








p(x) = 


E[X]=m 


2.3-9 
VAR [X] =o? ( ) 


A Gaussian random variable with m = 0 and o = 1 is called a standard normal. A 
function closely related to the Gaussian random variable is the Q function defined as 


O(x) = P[N(0, 1) > x] =F [ ao dt (2.3-10) 





FIGURE 2.3-1 
PDF and CDF of a Gaussian random variable. 


Px(x) = = err 
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=4 —3 —! 0 1 2 3 











4 yp —4 -3 is 
(b) (c) 
The CDF of a Gaussian random variable is given by 
x 1 (t—m 2 
F(x) = e 2.2 at 
-co V 210? 
1 / . s — dt 
= = e 20 
line 52 
a eae (2.3-11) 
1 u2 
= [= e 2 du 
x—m In 
-1-9 x—m 





where we have introduced the change of variable u = (¢ — m)/o. The PDF and the 
CDF of a Gaussian random variable are shown in Figure 2.3-1. 
In general if X ~ N(m, o”), then 


a-m 
PIX > a= 0( ) 


Oo 











(2.3-12) 
m—a 
P[X <a]=Q ( ) 
a 
Following are some of the important properties of the Q function: 
1 
OO= 5 Q(co) = 0 (2.3-13) 
Q(—o0) = 1 O(—x) = 1— Q(x) (2.3-14) 
1 2 
x ~ 2 2.3-16 
Q(x) me ( ) 





The distribution of a normal random variable with mean 0 and variance 1 is called 
a standard normal distribution. 

Creating a new random variable by this transformation is referred to as standardizing. 
The random variable Z represents the distance of X from its mean in terms of standard 


deviations. It is the key step to calculating a probability for an arbitrary normal random 
variable. 
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Standardizing 
to Calculate 


Suppose X is a normal random variable with mean ys. and variance o*. Then, 
a Probability 


P(X<x= (25 a) =AZ=<2 (4-11) 


xX — 
where Z is a Standard normal random variable, and z = ( ss b) is the z-value 


obtained by standardizing X. The probability is obtained by using Appendix Table III 
with z= (x — p)/o. 





3-8 EXPONENTIAL DISTRIBUTION 
Exponential 


Distribution The random variable X that equals the distance between successive events of a Poisson 


process with mean number of events A > 0 per unit interval is an exponential 
random variable with parameter A. The probability density function of X is 


fx) =Ae™ for 0=x¥< & (4-14) 





Mean and 


Variance If the random variable X has an exponential distribution with parameter A, 


1 


w= AX) => and o® = WN) =5 





3.9 The Central Limit Theorem 


What it the central limit theorem? The theorem says that under rather gen- 
eral circumstances, if you sum independent random variables and normalize 
them accordingly, then at the limit (when you sum lots of them) you'll get a 
normal distribution. 
For reference, here is the density of the normal distribution NV(j1, 07) with 
mean jt and variance 0: 
1 _ (=p)? 
(e-n)* 


———( 2a 
V 2ro? 


We now state a very weak form of the central limit theorem. Suppose that 


X; are independent, identically distributed random variables with zero mean 
and variance a7. Then 


Ky to 2, 
ATT N(0, 0). 
As the sample size n increases, the distribution of the sample mean wu of a random 


sample from a population (not necessarily normal) with mean w and variance o7 
approaches normal with mean u and variance o7/n. 
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Central Limit Theorem: Suppose Xj,...,Xn are 
i.i.d. with ELX,]) = mw and Var(X;) = o?. Then as 
nm o, 
Di Ai — NM i Xp D 
avn a/ Jn 
where 2" means that the c.d.f. — the Nor(O, 1) 


C.T. 
Remarks: (1) So if n is large, then X = Nor(y, 0?/n). 


Nor(0O, 1), 


(2) The X;'s don’t have to be normal for the CLT to 


work! 


(3) You usually need n > 30 observations for the ap- 
proximation to work well. (Need fewer observations if 
the X;'s come from a symmetric distribution. ) 


(4) You can almost always use the CLT if the obser- 


vations are i.i.d. 

The normal approximation for X will generally be good if n > 30, provided 
the population distribution is not terribly skewed. If n < 30, the approximation is 
good only if the population is not too different from a normal distribution and, as 
stated above, if the population is known to be normal, the sampling distribution 
of X will follow a normal distribution exactly, no matter how small the size of the 
samples. 

The sample size n = 30 is a guideline to use for the Central Limit Theorem. 
However, as the statement of the theorem implies, the presumption of normality 
on the distribution of X becomes more accurate as n grows larger. In fact, Figure 
8.1 illustrates how the theorem works. It shows how the distribution of X becomes 
closer to normal as n grows larger, beginning with the clearly nonsymmetric dis- 
tribution of an individual observation (n = 1). It also illustrates that the mean of 
X remains j: for any sample size and the variance of X gets smaller as n increases. 
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Large n (near normal) 


n=1 (population) 








Figure 8.1: Illustration of the Central Limit Theorem (distribution of X for n = 1, 
moderate n, and large n). 
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0.16 
0.68 SS 
0.12 | 





j= 


/3 (648 


Mi. 


n=5 


E&CE 411, Spring 2009, Table of Q Function 


Table 1: Values of Q(x) for0 <a <9 





























£ Q(z) £ Q(z) r Q(z) £ Q(z) 
0.00 0.5 2.30 0.010724 4.55 2.6823x10-° || 6.80 5.231x10-” 
0.05 0.48006 || 2.35 0.0093867 || 4.60 2.1125x10-® | 6.85 3.6925x10-!? 
0.10 0.46017 || 2.40 0.0081975 || 4.65 1.6597x10-® | 6.90 2.6001x10~!? 
0.15 0.44038 || 2.45 0.0071428 4.70 1.3008x10-© || 6.95 1.8264 107! 
0.20 0.42074 || 2.50 0.0062097 || 4.75 1.0171x10~® | 7.00 1.2798x10~!? 
0.25 0.40129 || 2.55 0.0053861 || 4.80 7.9333x10-" || 7.05 8.9459x107!8 
0.30 0.38209 || 2.60 0.0046612 || 4.85 6.1731x10-7 || 7.10 6.2378x10~!% 
0.35 0.36317 || 2.65 0.0040246 || 4.90 4.7918x107" | 7.15 4.3389x107!8 
0.40 0.34458 |] 2.70 0.003467 4.95 3.7107x10-" || 7.20 3.0106x10~-!8 
0.45 0.32636 || 2.75 0.0029798 || 5.00 2.8665x10~% || 7.25 2.0839x107!3 
0.50 0.30854 || 2.80 0.0025551 || 5.05 2.2091x10-7 || 7.30 1.4388x10~!% 
0.55 0.29116 || 2.85 0.002186 5.10 1.6983x10~" | 7.35 9.9103x107~4 
0.60 0.27425 || 2.90 0.0018658 || 5.15 1.3024x10-7 || 7.40 6.8092x10-!4 
0.65 0.25785 || 2.95 0.0015889 || 5.20 9.9644x10~§ || 7.45 4.667x10~4 
0.70 0.24196 |} 3.00 0.0013499 || 5.25 7.605x10-® | 7.50 3.1909x10~-!4 
0.75 0.22663 || 3.05 0.0011442 || 5.30 5.7901x10~® || 7.55 2.1763x107!4 
0.80 0.21186 |} 3.10 0.0009676 || 5.35 4.3977x10-® | 7.60 1.4807x10~!4 
0.85 0.19766 || 3.15 0.00081635 || 5.40 3.332x107-8 || 7.65 1.0049x10~-!4 
0.90 0.18406 || 3.20 0.00068714 || 5.45 2.5185x107® | 7.70 6.8033x1071° 
0.95 0.17106 || 3.25  0.00057703 || 5.50 1.899x10-8 | 7.75 4.5946x1071° 
1.00 0.15866 |) 3.30 0.00048342 |) 5.55 1.4283x107® | 7.80 3.0954x10~' 
1.05 0.14686 |) 3.35 0.00040406 || 5.60 1.0718x10-8 | 7.85 2.0802x10~-! 
1.10 0.13567 || 3.40 0.00033693 || 5.65 8.0224x10-® | 7.90 1.3945x107-' 
1.15 0.12507 || 3.45  0.00028029 |) 5.70 5.9904x10-9 || 7.95 9.3256x10~!6 
1.20 0.11507 || 3.50 0.00023263 |) 5.75 4.4622x10~-° || 8.00 6.221x107!6 
1.25 0.10565 || 3.55 0.00019262 || 5.80 3.3157x10~9 || 8.05 4.1397x10~!6 
1.30 0.0968 || 3.60 0.00015911 |) 5.85 2.4579x107-® || 8.10 2.748x107!6 
1.35. 0.088508 |) 3.65 0.00013112 || 5.90 1.8175x10~9 || 8.15 1.8196 10~!6 
1.40 0.080757 |} 3.70 0.0001078 5.95 1.3407x10-° |} 8.20 1.2019x10~16 
1.45 0.073529 || 3.75 8.8417x107° || 6.00 9.8659x107~!9 || 8.25 7.9197x1071!” 
1.50 0.066807 || 3.80 7.2348x10-° || 6.05 7.2423x107!° || 8.30 5.2056x10—!7 
1.55 0.060571 || 3.85 5.9059x107> |] 6.10 5.3034x107!° || 8.35 3.4131« 107!" 
1.60 0.054799 || 3.90 4.8096x10-° ] 6.15 3.8741x107!9 || 8.40 2.2324x107-!" 
1.65 0.049471 || 3.95 3.9076x10-° |] 6.20 2.8232x107!9 || 8.45 1.4565x1071!” 
1.70 0.044565 || 4.00 3.1671x10~° |] 6.25 2.0523x107!9 | 8.50 9.4795x107!8 
1.75 0.040059 |} 4.05 2.5609x107° || 6.30 1.4882x107!9 || 8.55 6.1544«1071!8 
1.80 0.03593 || 4.10 2.0658x10-° |) 6.35 1.0766x10~!° || 8.60 3.9858 10718 
1.85 0.032157 || 4.15 1.6624x10-° || 6.40 7.7688x10~" || 8.65 2.575x107!8 
1.90 0.028717 || 4.20 1.3346x10-° |} 6.45 5.5925x10-!! || 8.70 1.6594x107!8 
1.95 0.025588 || 4.25 1.0689x10-° | 6.50 4.016x10~!! || 8.75 1.0668x10~!8 
2.00 0.02275 || 4.30 8.5399x107®° || 6.55 2.8769x107!! | 8.80 6.8408x107!9 
2.05 0.020182 |) 4.35 6.8069x10~® || 6.60 2.0558x10-!! | 8.85 4.376x107!9 
2.10 0.017864 || 4.40 5.4125x107© || 6.65 1.4655x107!! | 8.90 2.7923x107-!9 
2.15 0.015778 |] 4.45 4.2935x10-® || 6.70 1.0421x10-" | 8.95 1.7774x1071!9 
2.20 0.013903 || 4.50 3.3977x107® || 6.75 7.3923x107-!? | 9.00 1.1286x107!9 
2.25 0.012224 
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The Random and the Fixed 


¢ Probability is a lack of knowledge. 


¢ Randomly varying or uncertain events may still have some underlying 
characteristics that are “fixed” 


¢ Some of these can be 


If this does not change over time then the 


Probability distribution random process is strictly stationary 


vo 


Mean. Variance, Covariance «<—— If these do not “vary” over time then the 
random process is weakly stationary 
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What is a random process? 


¢ Arandom process is a series of random events 


¢ Can be discrete (student heights) or continuous (voice) or discretized 
(digitized voice) 


x Height of next student 
ee who enters the room 
Random 
Event/Variable 
Heights of next eight 
—— [X, X, X3 Xy Xs Xe Xy Xg students who enter the room 
Random Process 
(discrete) 
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But first ... 


¢ Let us be clear about what we mean by a random “process” 


¢ Arandom process is a series of random events 


¢ Can be discrete (student heights) or continuous (voice) or discretized 
(discretized voice) 


X(t) Human speech 


4 


Random Process 
(continuous) 





—_, [X(ty) X(t.) X(tg) X(ty) X(ts) X(te) X(ty) X(tg)] Samples taken at 


times t1, t2, ..., t8. 
Random Process 
(discrete) EE 302 - Probabilistic Methods in Electrical Engineering / Dr. Naveed R. Butt @ Jouf University 


Our friends in an uncertain world! 


¢ Random or uncertain events may still have some underlying 
characteristics that are “fixed” 


¢ Some of these can be 


Probability distribution 


v 


Mean, Variance, Covariance 
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Our friends in an uncertain world! 


¢ Random or uncertain events may still have some underlying 
characteristics that are “fixed” 


¢ Some of these can be 


nee aa ; If this does not change over time then the 
Probability distribution random process is strictly stationary 


v 


Mean. Variance, Covariance <—— If these do not “vary” over time then the 
random process is weakly stationary 
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Revision - Distribution? 


¢ A distribution is a collection of probabilities we assign to random 
events. 


¢ Collection? Graph, table, function ; 
1/6 
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Our friends in an uncertain world! 


¢ Random or uncertain events may still have some underlying 
characteristics that are “fixed” 


¢ Some of these can be 


= 2. . If this does not change over time then the 
Probability distribution random process is strictly stationary 


v 


Mean. Variance, Covariance <—— If these do not “vary” over time then the 
, random process is weakly stationary 
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Revision: Mean/Average/Expectation 


Mean = Center of mass for the distribution (balancing point) 


Number of students 





pe ee 


Student Marks 
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Our friends in an uncertain world! 


¢ Random or uncertain events may still have some underlying 
characteristics that are “fixed” 


¢ Some of these can be 


= 2. . If this does not change over time then the 
Probability distribution random process is strictly stationary 


vo 


Mean. Variance. Covariance «<—— If these do not “vary” over time then the 
, random process is weakly stationary 
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Revision: Variance = degree of spread (how much 
variation is there in the data/outcomes?) 


Medium variance 


ahaa 


High variance 


Chocee Sp 
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Low variance 





Our friends in an uncertain world! 


¢ Random or uncertain events may still have some underlying 
characteristics that are “fixed” 


¢ Some of these can be 


= 2. . If this does not change over time then the 
Probability distribution random process is strictly stationary 


vo 


Mean. Variance, Covariance <——— If these do not “vary” over time then the 
random process is weakly stationary 
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Covariance = degree of linear relationship 
between data/outcomes 


4000 


3500 


Can knowledge of one 
random process help us say 
something about the value of 
another? 


3000 


AC Power [W] 
no 
2 


Yes — if their covariance is - 
. 
2014 
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Air conditioner 


/™» 


Time 


—— Outdoor temperature 


40 


[ \ \35 


Temperature [°C] 
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Covariance = degree of linear relationship 
between data/outcomes 


2200 
Can knowledge of one sant L= 76.11(T-30) +1222.3 


° 
random process help us say R? = 0.784 


® 








1800 
something about the value of econ 
? 
another: ann 
y ¢ thei ; ; 1200 
ee their covariance is —_ | 
high! 30.0 32.0 34.0 36.0 38.0 40.0 


One way of checking linear relationship (“covariance”) is to plot 
the two variables against each other. 
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Covariance = degree of linear relationship 


between data/outcomes 


ea Abdallah’s Marks Faisal’s Marks 


Quiz 1 
Quiz 2 
Quiz 3 
Quiz 4 
Quiz 5 
Quiz 6 
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Covariance = degree of linear relationship 
between data/outcomes 


Abdallah gets high marks 
When Abdallah eels high even when Faisal gets low 
marks, so does Faisal — marks — no linear relation 
high linear relation 


Faisal’s Marks 
Faisal’s Marks 








Abdallah’s Marks Abdallah’s Marks 
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Covariance = degree of linear relationship 
between data/outcomes 


Abdallah gets high marks 

even when Faisal’s marks ? 
are low — no linear 

relation 


Faisal’s Marks 
Faisal’s Marks 








Abdallah’s Marks Abdallah’s Marks 
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Covariance = degree of linear relationship 
between data/outcomes 


¢ Covariance helps us understand if two random processes are 
Statistically related or not 
¢ Statistically independent processes have zero covariance 


¢ We usually normalize covariance so that it lies between -1 and 1 
¢ Normalized covariance is called “Correlation” 
¢ Noise in communications is mostly assumed to be independent of the 
message signal (i.e., we often assume zero correlation between signal 
and noise) 


How do we write 


Covariance of two 
random variables X and Y 


Independent random variables 
have zero covariance 


For a random process we use the 
autocovariance function 


“covariance” mathematically ? 


C(X,Y)= E[(X-E[X])(Y-E[Y])] 


E[XY]- uyby 


C(X,Y) = 0 


Cyx(ty, to) = ELA) — Bx) A) ~ Hx) 


a Covariance between two 


samples of random process X(t) 


EE 302 - Probabilistic Methods in Electrical Engineering / Dr. Naveed R. Butt @ Jouf University 19 


Mean and Autocovariance of a Weakly 
Stationary Process 


The mean of a weakly 
stationary process does 
not change with time 


The autocovariance function of 
a weakly stationary process 
depends only on the time 
difference between the samples 
and NOT on their actual values 


y(t) = Hy  forallt 


Cyy (to — t+) for all Ty and [5 


or 


Cyx (T) T=tg2—-ty 
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Calculating the autocorrelation function 


Ryx (ty, tz) = E[X(t,)X (t2)] 


$$ a 
T=t,-t, 


X(t) X(t) 
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Calculating the autocorrelation function 
¢ For a weakly stationary process X(t), we have 
Ryx (ty, tz) = Ryx(t2 — t1) = Rxx() 


Ryy(t) = EL[X(t+ 7)X(t)] 


$$. $ N $$ sa a 
T=t,-t, 








X(t) X (tp) = X(t, + T) 
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Mean and Autocorrelation of a Weakly 
Stationary Process 


The mean of a weakly [ly(t) — fly for qll t 


stationary process does 
not change with time 


The autocorrelation function of if i) — Cy — Cy —— C3 th en 

a weakly stationary process 

depends only on the time = — __ 
difference between the samples Ry xX (t> ty ) 7 Ry xX (ty C3 ) 


and NOT on their actual values 
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Autocovariance vs. Autocorrelation of a weakly 
stationary process 


Autocovariance Cyy (tT) =E[(X(t ae T) - Lx) (X(t) a Ly ) | 


function 
Oa Covariance between two 


samples of random process X(t) 
Only a minor difference 
between the two. 


Conceptually very similar. Correlation between two 


a samples of random process X(t) 


Autocorrelation Ryy (tT) =E[X(t + T)X (t)] 


function 


Important properties of the autocorrelation 


function 


Ryy(0) = E[X'(t)] 





It can be used to calculate 
power of the signal 
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It is symmetric 





Ryx(7)| S Ryx(0) 


MS 


At no point it exceeds 
the average power 
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Ergodic Processes 






Random Processes 







s¢ Slationary 


Stationary 


Ergodic 
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Ergodic Processes 







X(t) 
(4 ANN wa, 
oO ~~ ~* ee 
i A stationary random process is said 
: Xq(t,)| Statistics along realizations to be “ergodic in the mean” if 
Xp(t) 
mr (VY A / \ Cees 5 ensamble mean = time mean 
0 N \A™ \/ 
Statistics along time 
i AM wa 
Pina Ww {_eN t 
° LY 
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Questions??? Thoughts? ? 


° 


e 
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Fourier Transform 
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Spectra — the Ghosts in Your Signal 


ax 





Q. Can we write signals as sums of 
periodic functions (frequencies)? 


DIK 

. WAANA 
—> WWMM 
\/ 





Baking a Fourier Cake 


¢ Given: Signal shape (time-domain) 
¢ Ingredients: Sinusoids of different frequencies 
¢ Choose: How much of the each ingredient (sinusoid) to use? 





¢ In Fourier Transform, we want to look at signals in terms of a fixed set 
of ingredients 
¢ Ingredients : Sinusoids of different frequencies 





¢ In Fourier Transform, we want to look at signals in terms of a fixed set 
of ingredients 
¢ Ingredients : Sinusoids of different frequencies 


Ingredient Amount 
(sinusoid frequency) (scaling) 
1 Add all 


0.5 
0.25 





¢ How is this shown after Fourier transform? 


Scaling 
Ingredient Amount 
+ (sinusoid frequency) at 
5 Hz Add all 
10 Hz 0.5 
+ 15 Hz 0.25 a 
0.25 


5 10 15 
Frequency (Hz) 


¢ We mostly skip the middle steps 


Amplitude 





Time 


Fourier Transform 


Inverse 
Fourier Transform 


Scaling 


0.5 
0.25 


5 10 15 
Frequency (Hz) 


AMPLITUDE 


F(t) 


TIME 


AMPLITUDE 





11 





Time Domain 


s(t) 





Frequency Domain 
S(w) 





14 


10 


16 


Time 


26 





O68 


o4 


os 


02 


280 300 360 


Frequency 


15 


Power Spectrum 
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Frequency content of a random process 


e We’ve seen that 
¢ Fourier Transform can be used to see the 
“frequency content” of a signal 
¢ But what if the signal is random? 


¢ Problem: frequency content may change 
from one realization to another! 


_——* X(t) = Acos(wt + @). 


A and ¢ random variables 





Autocorrelation Function to the rescue!! 


¢ We've seen that 
¢ Fourier Transform can be used to see the “frequency content” of a signal 


But what if the signal is random? 
¢ Problem: frequency content may change from one realization to another! 


¢ Two important observations can help us out 
¢ 1. The autocorrelation function of a weakly stationary process remains “fixed” between 
realizations 
e 2. The autocorrelation function contains the same frequencies as the original signal (with 
“average power” scalings) 


¢ Solution: 
¢ For a random process it is better to take the Fourier Transform of the covariance function 


For a random process it is better to take the Fourier 
Transform of the autocorrelation function!! 


¢ Power Spectrum = Fourier Transform of the autocorrelation function 


¢ Mathematically speaking ... 
Syy(f) = | _Ryx(2) exp(-j2afr) de 


CO 


Ryy( t) = | | Syy(f) exp(j2afr) df 
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Recall... 


Ryx(t) =E[X(¢ + 1)X(t)] 
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am plitude 


1000 


flute 


2000 
frequency (Hz) 





3000 





recorder 
25 923 
a 20 
2 15 
G10 
5 
= 2770 3694 
0 
0 1000 2000 3000 4000 
frequency (H2) 


——=r= 
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Spectral Magnitude (dB) 


Amplitude 


0.5 


0 


! ! 
Fi F2 





5000 On 


F2 FS Fars 


Frequency (Hz) 


FS FaFS 
Frequency (Hz) 


05 _ 0.01 0015 oo oO 0.005 _0.01 0.015 


Time ts) Time (S$) 


Voice Recognition 


5000 


0.02 
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Linear Systems 
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What happens to a signal as it passes through 
a systems 


¢ We shall only consider a special type of systems called LTI (linear 
time-invariant systems). 


¢ We shall only consider weakly stationary signals. 


Excitation Linear system: Response 
h(t) , 
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What happens to a signal as it passes through 
a systems 


¢ An LTI system is fully characterized by its Impulse Response (IR) 


¢ IR = what output the system gives when the input is an impulse (a theoretical 
sharp pulse) 


| 
——>» Channel ——> h(t) 
— 
Impulse Impulse response 
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What happens to a signal as it passes through 
a systems 


¢ If we know the impulse response of an LTI system we can find its 
output to any signal by using convolution 


y(t) = | h(t)x(t— tT) dt 


Excitation Linear system: Response 
i — impulse response —E ee 
hit) , 
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What happens to a signal as it passes through 
a system? 


¢ LTI system can also be fully characterized by its Transfer Function (TF) 
¢ TF = Fourier Transform of Impulse Response 


¢ TF = what the system does to different frequencies of the signal (e.g., blocks, 
allows unchanged, amplifies, attenuates) 


. Transfer Function output Frequencies 
Input Frequencies 


Amplified 


VAN AN A os adore ee 

















fh fe fs fa fe fe fi fr fs fa fs fe fi fa fs fe fs fe 
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What happens to a signal as it passes through 
a system? 


¢ If we know the Transfer Function of an LTI system we can find what it 
does to the frequencies of a deterministic signal by using simple 
multiplication in frequency domain 


YP) = AP)XGD) 


LT! System 


Input signal Output signal 


Y(f)=X(F)H(F) 


Transfer Function =f 
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What happens to a signal as it passes through 
a system? 


¢ If we know the Transfer Function of an LTI system we can find what it 
does to the power spectrum of weakly stationary random signal by 


? . 
Syyxf) = IACI Sy) IAP = HPH*(f) 


Questions??? Thoughts? ? 


° 


e 
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